SpySMAC

Plot default versus incumbent performance on a cumulative distribution plot. Uses validated data!

Plot all evaluated instances on a scatter plot, to directly compare performance of incumbent and default for each instance. Uses validated data!

Using PCA to reduce dimensionality of the search space and plot the distribution of evaluated configurations. The bigger the dot, the more often the configuration was evaluated. The colours refer to the predicted performance in that part of the search space.

The cost of the incumbent estimated over the time. The cost is estimated using an EPM that is based on the actual runs.

Plot explored range of most important parameters.

fANOVA stands for functional analysis of variance and predicts a parameters marginal performance, by analyzing the predicted local neighbourhood of this parameters optimized value, considering correlations to other parameters and isolating this parameters importance by predicting performance changes that depend on other parameters.

Violin and Box plots to show the distribution of each instance feature. We removed NaN from the data.

Correlation based on Pearson product-moment correlation coefficients between all features and clustered with Wards hierarchical clustering approach. Darker fields corresponds to a larger correlation between the features.

Feature importance calculated using forward selection.

Clustering instances in 2d; the color encodes the cluster assigned to each cluster. Similar to ISAC, we use a k-means to cluster the instances in the feature space. As pre-processing, we use standard scaling and a PCA to 2 dimensions. To guess the number of clusters, we use the silhouette score on the range of 2 to 12 in the number of clusters